A Statistical Study on On-line Learning

In this paper we examine on-line learning with statistical framework. Firstly we study the cases with xed and annealed learning rate. It can be shown that on-line learning with 1=t annealed learning rate minimizes the generalization error with the same rate as batch learning in the asymp-totic regime, that is, on-line learning can be as eeective as batch learning asymptotically. Using these analyses, we study an adaptive learning rate algorithm which is based on Sompolinsky-Barkai-Seung algorithm and which achieves 1=t-annealing automatically. (1) which measures the performance of learning system for given input x and desired output y. (2) where determines the strength of regularization. We assume that loss d(x; y;) is diierentiable up to the appropriate order and we use r and rr to indicate 1